I have a suggestion. You can grab tool call action by tool token ids instead of token string.

In my own "Python implementation", if start tool call token id is generated(<|python_tag|> in case of llama 3.1) then streaming is paused until stop token/end_tool_call token is generated. Then a recursive call is made with the output of the tool and streaming is resumed.

@qnixsynapse Yes it's possible to do so with Hermes-3 format, but that will not be possible with either llama 3.1 JSON tool calls or llama 3.1 custom function. The goal here is to make it compatible with OAI specs, so relying on <|python_tag|> is not an option here (it may make more sense with llama-cpp-python)

Anyway, I'll consider doing this later on, when tool call templates are more mainstream and patterns start to emerge.

@ngxson The <|python_tag|> was just an example. You can also add support for newer mistral models(7B v3; Nemo) which has [TOOL_CALL] tokens for example.

Here for example, we can expand this:

else if (has_token("[/INST]") && has_token("[TOOL_CALLS]")) {
        return LLAMA_TOOL_FORMAT_MISTRAL;
 }

Regarding streaming, this is sufficient I think:

@ngxson, I appreciate your work, the new feature is great.
I've encountered an issue, though. When the conversation includes previous model messages with a "null" content (which can happen with tool_call only messages), I receive an "Invalid 'content' type" error on subsequent requests. I believe a minor adjustment to the parse_chat_messages function in /examples/server/utils.hpp should solve this.
For now, I've worked around the problem by manually assigning an empty string to "content" in the code that makes the calls to the server.

EDIT: Also the cases in which the "tools" parameter in the /completion request is null, or an empty array, should be accepted and treated the same as the case in which such parameter does not exist at all.